{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Store using FEAST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feast (Feature Store) is a tool for managing and serving machine learning features.\n",
    "\n",
    "To execute this notebook, you'll first need to install Feast and connect to an existing deployment. \n",
    "To get started, follow the instructions in this Getting Started guide [here](https://docs.feast.dev/getting-started). \n",
    "In short, you will need to:\n",
    "\n",
    " 1. Install [docker-compose](https://docs.docker.com/compose/install/) if it's not already on your machine. You can also deploy Feast using GKE which is better suited for production purposes.\n",
    " 1. Install feast\n",
    "     - ```python\n",
    "     pip install feast\n",
    "     ```\n",
    " 2. Clone the Feast repository and navigate to the infra/docker-compose sub-directory\n",
    "     - ```python\n",
    "     git clone https://github.com/feast-dev/feast.git\n",
    "     ```\n",
    "     - ```python\n",
    "     cd feast/infra/docker-compose\n",
    "     ```\n",
    " 3. Make a copy of the .env.sample file\n",
    "     - ```python\n",
    "     cp .env.sample .env\n",
    "     ```\n",
    " 4. Create a [service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) and copy it to the `infra/docker-compose/gcp-service-accounts` folder\n",
    " 5. Create a GCS bucket to use for staging\n",
    "      - ```bash \n",
    "      gsutil mb gs://my-feast-staging-bucket\n",
    "      ```\n",
    " 6. Configure the `.env` file to reference your service key:\n",
    "      - `FEAST_CORE_GCP_SERVICE_ACCOUNT_KEY`\n",
    "      - `FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY`\n",
    "      - `FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY`\n",
    "      \n",
    " 7. Configure the following fields in the `feast/infra/docker-compose/serving/batch-serving.yml` file:\n",
    "      - `feast.stores.config.project_id`\n",
    "      - `feast.stores.config.dataset_id`\n",
    "      - `feast.stores.config.staging_location`\n",
    "      \n",
    " 8. Start Feast:\n",
    "      ```bash\n",
    "      docker-compose \\\n",
    "        -f docker-compose.yml \\\n",
    "        -f docker-compose.online.yml \\\n",
    "        -f docker-compose.batch.yml \\\n",
    "        up -d\n",
    "      ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "Set up the serving clients for offline and batch feature retrieval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Feast Core acts as the central feature registry\n",
    "FEAST_CORE_URL = os.getenv('FEAST_CORE_URL', 'localhost:6565')\n",
    "\n",
    "# Feast Online Serving allows for the retrieval of real-time feature data\n",
    "FEAST_ONLINE_SERVING_URL = os.getenv('FEAST_ONLINE_SERVING_URL', 'localhost:6566')\n",
    "\n",
    "# Feast Batch Serving allows for the retrieval of historical feature data\n",
    "FEAST_BATCH_SERVING_URL = os.getenv('FEAST_BATCH_SERVING_URL', 'localhost:6567')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries and modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install --user feast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install --user xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pytz import timezone, utc\n",
    "from feast import Client, FeatureSet, Entity, ValueType\n",
    "from feast.serving.ServingService_pb2 import GetOnlineFeaturesRequest\n",
    "from feast.types.Value_pb2 import Value as Value\n",
    "from google.protobuf.duration_pb2 import Duration\n",
    "from datetime import datetime, timedelta\n",
    "from random import randrange\n",
    "import random\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from xgboost import XGBRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting features into Feast\n",
    "\n",
    "### Read in taxifare data\n",
    "Read in the taxifare data from .csv file and inspect with  Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.5,2020-04-01 22:32:37,-73.99022674560547,40.749820709228516,-73.99752807617188,40.744781494140625,4,0\n",
      "5.0,2020-06-14 00:21:24,-73.9786605834961,40.78569030761719,-73.9762191772461,40.77631378173828,1,0\n",
      "5.0,2020-02-28 10:50:48,-73.99897003173828,40.745609283447266,-73.9891357421875,40.747825622558594,1,0\n",
      "8.5,2020-01-20 16:25:38,-73.987548828125,40.72443389892578,-73.977783203125,40.7481803894043,1,0\n",
      "6.0,2020-03-04 19:24:49,-73.98148345947266,40.76824188232422,-73.98226165771484,40.75939178466797,3,0\n",
      "7.0,2020-08-23 11:36:04,-73.93275451660156,40.79555892944336,-73.94598388671875,40.777591705322266,1,0\n",
      "8.5,2020-04-13 19:24:08,-73.98275756835938,40.767696380615234,-73.98104095458984,40.78606414794922,1,0\n",
      "9.0,2020-04-07 12:31:59,-73.87153625488281,40.75267028808594,-73.88893127441406,40.748294830322266,1,0\n",
      "6.0,2020-09-05 18:36:26,-73.9574203491211,40.76603317260742,-73.95464324951172,40.7779426574707,1,0\n",
      "6.0,2020-02-24 23:28:34,-73.99248504638672,40.72425842285156,-73.98728942871094,40.73808670043945,1,0\n"
     ]
    }
   ],
   "source": [
    "!head taxi-train.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "COL_NAMES = ['fare_amount', 'pickup_datetime', 'pickup_longitude', \\\n",
    "             'pickup_latitude', 'dropoff_longitude', 'dropoff_latitude', 'passenger_count', 'taxi_id']\n",
    "\n",
    "taxi_df = pd.read_csv('taxi-train.csv', names=COL_NAMES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>pickup_datetime</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>taxi_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.5</td>\n",
       "      <td>2020-04-01 22:32:37</td>\n",
       "      <td>-73.990227</td>\n",
       "      <td>40.749821</td>\n",
       "      <td>-73.997528</td>\n",
       "      <td>40.744781</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2020-06-14 00:21:24</td>\n",
       "      <td>-73.978661</td>\n",
       "      <td>40.785690</td>\n",
       "      <td>-73.976219</td>\n",
       "      <td>40.776314</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2020-02-28 10:50:48</td>\n",
       "      <td>-73.998970</td>\n",
       "      <td>40.745609</td>\n",
       "      <td>-73.989136</td>\n",
       "      <td>40.747826</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.5</td>\n",
       "      <td>2020-01-20 16:25:38</td>\n",
       "      <td>-73.987549</td>\n",
       "      <td>40.724434</td>\n",
       "      <td>-73.977783</td>\n",
       "      <td>40.748180</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2020-03-04 19:24:49</td>\n",
       "      <td>-73.981483</td>\n",
       "      <td>40.768242</td>\n",
       "      <td>-73.982262</td>\n",
       "      <td>40.759392</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fare_amount      pickup_datetime  pickup_longitude  pickup_latitude  \\\n",
       "0          5.5  2020-04-01 22:32:37        -73.990227        40.749821   \n",
       "1          5.0  2020-06-14 00:21:24        -73.978661        40.785690   \n",
       "2          5.0  2020-02-28 10:50:48        -73.998970        40.745609   \n",
       "3          8.5  2020-01-20 16:25:38        -73.987549        40.724434   \n",
       "4          6.0  2020-03-04 19:24:49        -73.981483        40.768242   \n",
       "\n",
       "   dropoff_longitude  dropoff_latitude  passenger_count  taxi_id  \n",
       "0         -73.997528         40.744781                4        0  \n",
       "1         -73.976219         40.776314                1        0  \n",
       "2         -73.989136         40.747826                1        0  \n",
       "3         -73.977783         40.748180                1        0  \n",
       "4         -73.982262         40.759392                3        0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxi_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# needs datetime field in datetime[ns] format\n",
    "# create a datetime field from pickup_datetime\n",
    "taxi_datetime = pd.to_datetime(taxi_df.pickup_datetime, unit='ns', utc=True)\n",
    "taxi_df.insert(2, \"datetime\", taxi_datetime, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>pickup_datetime</th>\n",
       "      <th>datetime</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>taxi_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.5</td>\n",
       "      <td>2020-04-01 22:32:37</td>\n",
       "      <td>2020-04-01 22:32:37+00:00</td>\n",
       "      <td>-73.990227</td>\n",
       "      <td>40.749821</td>\n",
       "      <td>-73.997528</td>\n",
       "      <td>40.744781</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2020-06-14 00:21:24</td>\n",
       "      <td>2020-06-14 00:21:24+00:00</td>\n",
       "      <td>-73.978661</td>\n",
       "      <td>40.785690</td>\n",
       "      <td>-73.976219</td>\n",
       "      <td>40.776314</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2020-02-28 10:50:48</td>\n",
       "      <td>2020-02-28 10:50:48+00:00</td>\n",
       "      <td>-73.998970</td>\n",
       "      <td>40.745609</td>\n",
       "      <td>-73.989136</td>\n",
       "      <td>40.747826</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.5</td>\n",
       "      <td>2020-01-20 16:25:38</td>\n",
       "      <td>2020-01-20 16:25:38+00:00</td>\n",
       "      <td>-73.987549</td>\n",
       "      <td>40.724434</td>\n",
       "      <td>-73.977783</td>\n",
       "      <td>40.748180</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2020-03-04 19:24:49</td>\n",
       "      <td>2020-03-04 19:24:49+00:00</td>\n",
       "      <td>-73.981483</td>\n",
       "      <td>40.768242</td>\n",
       "      <td>-73.982262</td>\n",
       "      <td>40.759392</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fare_amount      pickup_datetime                  datetime  \\\n",
       "0          5.5  2020-04-01 22:32:37 2020-04-01 22:32:37+00:00   \n",
       "1          5.0  2020-06-14 00:21:24 2020-06-14 00:21:24+00:00   \n",
       "2          5.0  2020-02-28 10:50:48 2020-02-28 10:50:48+00:00   \n",
       "3          8.5  2020-01-20 16:25:38 2020-01-20 16:25:38+00:00   \n",
       "4          6.0  2020-03-04 19:24:49 2020-03-04 19:24:49+00:00   \n",
       "\n",
       "   pickup_longitude  pickup_latitude  dropoff_longitude  dropoff_latitude  \\\n",
       "0        -73.990227        40.749821         -73.997528         40.744781   \n",
       "1        -73.978661        40.785690         -73.976219         40.776314   \n",
       "2        -73.998970        40.745609         -73.989136         40.747826   \n",
       "3        -73.987549        40.724434         -73.977783         40.748180   \n",
       "4        -73.981483        40.768242         -73.982262         40.759392   \n",
       "\n",
       "   passenger_count  taxi_id  \n",
       "0                4        0  \n",
       "1                1        0  \n",
       "2                1        0  \n",
       "3                1        0  \n",
       "4                3        0  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxi_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create new features: Euclidean distance\n",
    "\n",
    "Engineer an additional feature which provides the Euclidean distance from pickup location to dropoff location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_dist(row):\n",
    "    lat1, lon1 = row.pickup_latitude, row.pickup_longitude\n",
    "    lat2, lon2 = row.dropoff_latitude, row.dropoff_longitude\n",
    "    londiff = lon2 - lon1\n",
    "    latdiff = lat2 - lat1\n",
    "    return np.sqrt(londiff*londiff + latdiff*latdiff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "taxi_df['euclid_dist'] = taxi_df.apply(compute_dist, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>pickup_datetime</th>\n",
       "      <th>datetime</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>taxi_id</th>\n",
       "      <th>euclid_dist</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.5</td>\n",
       "      <td>2020-04-01 22:32:37</td>\n",
       "      <td>2020-04-01 22:32:37+00:00</td>\n",
       "      <td>-73.990227</td>\n",
       "      <td>40.749821</td>\n",
       "      <td>-73.997528</td>\n",
       "      <td>40.744781</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0.008871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2020-06-14 00:21:24</td>\n",
       "      <td>2020-06-14 00:21:24+00:00</td>\n",
       "      <td>-73.978661</td>\n",
       "      <td>40.785690</td>\n",
       "      <td>-73.976219</td>\n",
       "      <td>40.776314</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.009689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2020-02-28 10:50:48</td>\n",
       "      <td>2020-02-28 10:50:48+00:00</td>\n",
       "      <td>-73.998970</td>\n",
       "      <td>40.745609</td>\n",
       "      <td>-73.989136</td>\n",
       "      <td>40.747826</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.010081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.5</td>\n",
       "      <td>2020-01-20 16:25:38</td>\n",
       "      <td>2020-01-20 16:25:38+00:00</td>\n",
       "      <td>-73.987549</td>\n",
       "      <td>40.724434</td>\n",
       "      <td>-73.977783</td>\n",
       "      <td>40.748180</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.025676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2020-03-04 19:24:49</td>\n",
       "      <td>2020-03-04 19:24:49+00:00</td>\n",
       "      <td>-73.981483</td>\n",
       "      <td>40.768242</td>\n",
       "      <td>-73.982262</td>\n",
       "      <td>40.759392</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.008884</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fare_amount      pickup_datetime                  datetime  \\\n",
       "0          5.5  2020-04-01 22:32:37 2020-04-01 22:32:37+00:00   \n",
       "1          5.0  2020-06-14 00:21:24 2020-06-14 00:21:24+00:00   \n",
       "2          5.0  2020-02-28 10:50:48 2020-02-28 10:50:48+00:00   \n",
       "3          8.5  2020-01-20 16:25:38 2020-01-20 16:25:38+00:00   \n",
       "4          6.0  2020-03-04 19:24:49 2020-03-04 19:24:49+00:00   \n",
       "\n",
       "   pickup_longitude  pickup_latitude  dropoff_longitude  dropoff_latitude  \\\n",
       "0        -73.990227        40.749821         -73.997528         40.744781   \n",
       "1        -73.978661        40.785690         -73.976219         40.776314   \n",
       "2        -73.998970        40.745609         -73.989136         40.747826   \n",
       "3        -73.987549        40.724434         -73.977783         40.748180   \n",
       "4        -73.981483        40.768242         -73.982262         40.759392   \n",
       "\n",
       "   passenger_count  taxi_id  euclid_dist  \n",
       "0                4        0     0.008871  \n",
       "1                1        0     0.009689  \n",
       "2                1        0     0.010081  \n",
       "3                1        0     0.025676  \n",
       "4                3        0     0.008884  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxi_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to FEAST and create a FeatureSet with this dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to FEAST core\n",
    "client = Client(core_url=FEAST_CORE_URL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.list_feature_sets()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "FS_NAME = \"taxirides\"\n",
    "taxi_fs = FeatureSet(\"taxirides\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entity taxi_id(ValueType.INT64) manually updated (replacing an existing field).\n",
      "Feature fare_amount (ValueType.DOUBLE) added from dataframe.\n",
      "Feature pickup_longitude (ValueType.DOUBLE) added from dataframe.\n",
      "Feature pickup_latitude (ValueType.DOUBLE) added from dataframe.\n",
      "Feature dropoff_longitude (ValueType.DOUBLE) added from dataframe.\n",
      "Feature dropoff_latitude (ValueType.DOUBLE) added from dataframe.\n",
      "Feature passenger_count (ValueType.INT64) added from dataframe.\n",
      "Feature euclid_dist (ValueType.DOUBLE) added from dataframe.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "taxi_fs.infer_fields_from_df(taxi_df,\n",
    "                             entities=[Entity(name='taxi_id', dtype=ValueType.INT64)],\n",
    "                             replace_existing_features=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature set created: \"taxirides\"\n"
     ]
    }
   ],
   "source": [
    "client.apply(taxi_fs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[default/taxirides]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.list_feature_sets()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"spec\": {\n",
      "    \"name\": \"taxirides\",\n",
      "    \"entities\": [\n",
      "      {\n",
      "        \"name\": \"taxi_id\",\n",
      "        \"valueType\": \"INT64\"\n",
      "      }\n",
      "    ],\n",
      "    \"features\": [\n",
      "      {\n",
      "        \"name\": \"pickup_longitude\",\n",
      "        \"valueType\": \"DOUBLE\"\n",
      "      },\n",
      "      {\n",
      "        \"name\": \"dropoff_latitude\",\n",
      "        \"valueType\": \"DOUBLE\"\n",
      "      },\n",
      "      {\n",
      "        \"name\": \"dropoff_longitude\",\n",
      "        \"valueType\": \"DOUBLE\"\n",
      "      },\n",
      "      {\n",
      "        \"name\": \"fare_amount\",\n",
      "        \"valueType\": \"DOUBLE\"\n",
      "      },\n",
      "      {\n",
      "        \"name\": \"euclid_dist\",\n",
      "        \"valueType\": \"DOUBLE\"\n",
      "      },\n",
      "      {\n",
      "        \"name\": \"passenger_count\",\n",
      "        \"valueType\": \"INT64\"\n",
      "      },\n",
      "      {\n",
      "        \"name\": \"pickup_latitude\",\n",
      "        \"valueType\": \"DOUBLE\"\n",
      "      }\n",
      "    ],\n",
      "    \"maxAge\": \"0s\",\n",
      "    \"source\": {\n",
      "      \"type\": \"KAFKA\",\n",
      "      \"kafkaSourceConfig\": {\n",
      "        \"bootstrapServers\": \"kafka:9092,localhost:9094\",\n",
      "        \"topic\": \"feast-features\"\n",
      "      }\n",
      "    },\n",
      "    \"project\": \"default\"\n",
      "  },\n",
      "  \"meta\": {\n",
      "    \"createdTimestamp\": \"2020-06-26T18:45:45Z\",\n",
      "    \"status\": \"STATUS_PENDING\"\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(client.get_feature_set('taxirides'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for feature set to be ready for ingestion...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 28247/28247 [00:01<00:00, 19406.32rows/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingestion complete!\n",
      "\n",
      "Ingestion statistics:\n",
      "Success: 28247/28247\n",
      "Removing temporary file(s)...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'29c921f6-7462-3b73-92c6-50c02aeedc45'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.ingest(taxi_fs, taxi_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieving feature stores from Feast\n",
    "\n",
    "### Get batch features for training\n",
    "To access historical/office features, we'll set up a Feast serving batch client.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "_feast_batch_client = Client(serving_url=FEAST_BATCH_SERVING_URL,\n",
    "                             core_url=FEAST_CORE_URL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_features = ['pickup_latitude',\n",
    "                  'pickup_longitude',\n",
    "                  'dropoff_latitude',\n",
    "                  'dropoff_longitude',\n",
    "                  'passenger_count',\n",
    "                  'euclid_dist']\n",
    "\n",
    "target = 'fare_amount'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add the target variable to our feature list\n",
    "features = model_features + [target]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To pull batch features, we provide an `entity` dataframe that contains the entities and timestamps we want to retrieve. We'll provide every pairing to get all offline features for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "taxis = taxi_df.taxi_id.unique()\n",
    "days = taxi_df.datetime.unique() \n",
    "\n",
    "entity_df = pd.DataFrame(\n",
    "    {\n",
    "        \"datetime\": [day for day in days for taxi in taxis],\n",
    "        \"taxi_id\": [taxi for day in days for taxi in taxis],\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(109193, 2)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entity_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "FS_NAME = \"taxirides\"\n",
    "\n",
    "# Retrieve training dataset from Feast\n",
    "dataset = _feast_batch_client.get_batch_features(\n",
    "    feature_refs=[FS_NAME + \":\" + feature for feature in features],\n",
    "    entity_rows=entity_df).to_dataframe()\n",
    "dataset.dropna(inplace=True)  # not all pairing of datetime and taxi_id have entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>event_timestamp</th>\n",
       "      <th>taxi_id</th>\n",
       "      <th>taxirides__pickup_latitude</th>\n",
       "      <th>taxirides__pickup_longitude</th>\n",
       "      <th>taxirides__dropoff_latitude</th>\n",
       "      <th>taxirides__dropoff_longitude</th>\n",
       "      <th>taxirides__passenger_count</th>\n",
       "      <th>taxirides__euclid_dist</th>\n",
       "      <th>taxirides__fare_amount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1044</th>\n",
       "      <td>2020-04-10 14:31:50+00:00</td>\n",
       "      <td>0</td>\n",
       "      <td>40.75267</td>\n",
       "      <td>-73.871536</td>\n",
       "      <td>40.748295</td>\n",
       "      <td>-73.888931</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.017937</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1045</th>\n",
       "      <td>2020-04-08 10:59:21+00:00</td>\n",
       "      <td>0</td>\n",
       "      <td>40.75267</td>\n",
       "      <td>-73.871536</td>\n",
       "      <td>40.748295</td>\n",
       "      <td>-73.888931</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.017937</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1046</th>\n",
       "      <td>2020-04-09 06:13:34+00:00</td>\n",
       "      <td>0</td>\n",
       "      <td>40.75267</td>\n",
       "      <td>-73.871536</td>\n",
       "      <td>40.748295</td>\n",
       "      <td>-73.888931</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.017937</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1047</th>\n",
       "      <td>2020-04-12 14:27:28+00:00</td>\n",
       "      <td>0</td>\n",
       "      <td>40.75267</td>\n",
       "      <td>-73.871536</td>\n",
       "      <td>40.748295</td>\n",
       "      <td>-73.888931</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.017937</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048</th>\n",
       "      <td>2020-04-09 22:36:21+00:00</td>\n",
       "      <td>0</td>\n",
       "      <td>40.75267</td>\n",
       "      <td>-73.871536</td>\n",
       "      <td>40.748295</td>\n",
       "      <td>-73.888931</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.017937</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               event_timestamp  taxi_id  taxirides__pickup_latitude  \\\n",
       "1044 2020-04-10 14:31:50+00:00        0                    40.75267   \n",
       "1045 2020-04-08 10:59:21+00:00        0                    40.75267   \n",
       "1046 2020-04-09 06:13:34+00:00        0                    40.75267   \n",
       "1047 2020-04-12 14:27:28+00:00        0                    40.75267   \n",
       "1048 2020-04-09 22:36:21+00:00        0                    40.75267   \n",
       "\n",
       "      taxirides__pickup_longitude  taxirides__dropoff_latitude  \\\n",
       "1044                   -73.871536                    40.748295   \n",
       "1045                   -73.871536                    40.748295   \n",
       "1046                   -73.871536                    40.748295   \n",
       "1047                   -73.871536                    40.748295   \n",
       "1048                   -73.871536                    40.748295   \n",
       "\n",
       "      taxirides__dropoff_longitude  taxirides__passenger_count  \\\n",
       "1044                    -73.888931                         1.0   \n",
       "1045                    -73.888931                         1.0   \n",
       "1046                    -73.888931                         1.0   \n",
       "1047                    -73.888931                         1.0   \n",
       "1048                    -73.888931                         1.0   \n",
       "\n",
       "      taxirides__euclid_dist  taxirides__fare_amount  \n",
       "1044                0.017937                     9.0  \n",
       "1045                0.017937                     9.0  \n",
       "1046                0.017937                     9.0  \n",
       "1047                0.017937                     9.0  \n",
       "1048                0.017937                     9.0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test, y_train, y_test = \\\n",
    "    train_test_split(dataset[[FS_NAME + \"__\" + feature for feature in model_features]],\n",
    "                     dataset[FS_NAME + \"__\" + target],\n",
    "                     test_size=0.25, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "             colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "             importance_type='gain', interaction_constraints='',\n",
       "             learning_rate=0.1, max_delta_step=0, max_depth=3,\n",
       "             min_child_weight=1, missing=None, monotone_constraints='()',\n",
       "             n_estimators=100, n_jobs=1, nthread=1, num_parallel_tree=1,\n",
       "             objective='reg:squarederror', random_state=0, reg_alpha=0,\n",
       "             reg_lambda=1, scale_pos_weight=1, seed=0, silent=None, subsample=1,\n",
       "             tree_method='exact', validate_parameters=1, verbosity=1)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
    "       colsample_bynode=1, colsample_bytree=1, gamma=0,\n",
    "       importance_type='gain', learning_rate=0.1, max_delta_step=0,\n",
    "       max_depth=3, min_child_weight=1, missing=None, n_estimators=100,\n",
    "       n_jobs=1, nthread=None, objective='reg:squarederror', random_state=0,\n",
    "       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n",
    "       silent=None, subsample=1, verbosity=1)\n",
    "\n",
    "# Next, we'll fit the model with training data.\n",
    "model.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training score:  0.9999999890464406\n",
      "Testing score:  0.9999999891959381\n"
     ]
    }
   ],
   "source": [
    "train_score = model.score(x_train, y_train)  \n",
    "test_score = model.score(x_test, y_test)  \n",
    "print(\"Training score: \", train_score)\n",
    "print(\"Testing score: \", test_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict with online features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "_feast_online_client = Client(serving_url=FEAST_ONLINE_SERVING_URL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for a single taxi_id\n",
    "taxi_id = 1\n",
    "\n",
    "online_features = _feast_online_client.get_online_features(\n",
    "    feature_refs=[FS_NAME + \":\" + feature for feature in model_features],\n",
    "    entity_rows=[\n",
    "        GetOnlineFeaturesRequest.EntityRow(\n",
    "            fields={\n",
    "                \"taxi_id\": Value(\n",
    "                    int64_val=taxi_id)\n",
    "            }\n",
    "        )\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "field_values {\n",
      "  fields {\n",
      "    key: \"taxi_id\"\n",
      "    value {\n",
      "      int64_val: 1\n",
      "    }\n",
      "  }\n",
      "  fields {\n",
      "    key: \"taxirides:dropoff_latitude\"\n",
      "    value {\n",
      "      double_val: 40.78923797607422\n",
      "    }\n",
      "  }\n",
      "  fields {\n",
      "    key: \"taxirides:dropoff_longitude\"\n",
      "    value {\n",
      "      double_val: -73.96871948242188\n",
      "    }\n",
      "  }\n",
      "  fields {\n",
      "    key: \"taxirides:euclid_dist\"\n",
      "    value {\n",
      "      double_val: 0.015991973949749216\n",
      "    }\n",
      "  }\n",
      "  fields {\n",
      "    key: \"taxirides:passenger_count\"\n",
      "    value {\n",
      "      int64_val: 1\n",
      "    }\n",
      "  }\n",
      "  fields {\n",
      "    key: \"taxirides:pickup_latitude\"\n",
      "    value {\n",
      "      double_val: 40.77964401245117\n",
      "    }\n",
      "  }\n",
      "  fields {\n",
      "    key: \"taxirides:pickup_longitude\"\n",
      "    value {\n",
      "      double_val: -73.95592498779297\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(online_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to Pandas dataframe\n",
    "features_dict = dict.fromkeys([FS_NAME + \"__\" + feature for feature in model_features])\n",
    "\n",
    "for row in online_features.field_values:\n",
    "    for feature in model_features:\n",
    "        if features_dict[FS_NAME + \"__\" + feature] is None:\n",
    "            features_dict[FS_NAME + \"__\" + feature] = []    \n",
    "        if feature in ['passenger_count']:\n",
    "            features_dict[FS_NAME + \"__\" + feature].append(row.fields[FS_NAME + \":\" + feature].int64_val)\n",
    "        else:\n",
    "            features_dict[FS_NAME + \"__\" + feature].append(row.fields[FS_NAME + \":\" + feature].double_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'taxirides__pickup_latitude': [40.77964401245117],\n",
       " 'taxirides__pickup_longitude': [-73.95592498779297],\n",
       " 'taxirides__dropoff_latitude': [40.78923797607422],\n",
       " 'taxirides__dropoff_longitude': [-73.96871948242188],\n",
       " 'taxirides__passenger_count': [1],\n",
       " 'taxirides__euclid_dist': [0.015991973949749216]}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_df = pd.DataFrame.from_dict(features_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4.9998827], dtype=float32)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(predict_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch predict job for all taxi_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Pandas dataframe\n",
    "features_dict = dict.fromkeys([FS_NAME + \"__\" + feature for feature in model_features] + ['taxi_id'])\n",
    "\n",
    "# all taxi_ids\n",
    "taxi_ids = taxi_df.taxi_id.unique()\n",
    "\n",
    "entity_rows = []\n",
    "for taxi_id in taxi_ids.tolist():\n",
    "    entity_rows.append(\n",
    "        GetOnlineFeaturesRequest.EntityRow(fields={'taxi_id': Value(int64_val=taxi_id)})\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = _feast_online_client.get_online_features(\n",
    "    feature_refs=[FS_NAME + \":\" + feature for feature in model_features],\n",
    "    entity_rows=entity_rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "for row in data.field_values:\n",
    "    # capture taxi_id\n",
    "    if features_dict['taxi_id'] is None:\n",
    "        features_dict['taxi_id'] = []\n",
    "    features_dict['taxi_id'].append(row.fields['taxi_id'].int64_val)\n",
    "    \n",
    "    # get all feature values\n",
    "    for feature in model_features:\n",
    "        if features_dict[FS_NAME + \"__\" + feature] is None:\n",
    "            features_dict[FS_NAME + \"__\" + feature] = []    \n",
    "        if feature in ['passenger_count']:\n",
    "            features_dict[FS_NAME + \"__\" + feature].append(row.fields[FS_NAME + \":\" + feature].int64_val)\n",
    "        else:\n",
    "            features_dict[FS_NAME + \"__\" + feature].append(row.fields[FS_NAME + \":\" + feature].double_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>taxirides__pickup_latitude</th>\n",
       "      <th>taxirides__pickup_longitude</th>\n",
       "      <th>taxirides__dropoff_latitude</th>\n",
       "      <th>taxirides__dropoff_longitude</th>\n",
       "      <th>taxirides__passenger_count</th>\n",
       "      <th>taxirides__euclid_dist</th>\n",
       "      <th>taxi_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>40.748283</td>\n",
       "      <td>-73.985092</td>\n",
       "      <td>40.773758</td>\n",
       "      <td>-73.870667</td>\n",
       "      <td>1</td>\n",
       "      <td>0.117227</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40.722446</td>\n",
       "      <td>-73.987366</td>\n",
       "      <td>40.648792</td>\n",
       "      <td>-73.977425</td>\n",
       "      <td>1</td>\n",
       "      <td>0.074322</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>40.779644</td>\n",
       "      <td>-73.955925</td>\n",
       "      <td>40.789238</td>\n",
       "      <td>-73.968719</td>\n",
       "      <td>1</td>\n",
       "      <td>0.015992</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>40.752628</td>\n",
       "      <td>-74.004257</td>\n",
       "      <td>40.747871</td>\n",
       "      <td>-74.000458</td>\n",
       "      <td>2</td>\n",
       "      <td>0.006088</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>40.728329</td>\n",
       "      <td>-73.999672</td>\n",
       "      <td>40.761410</td>\n",
       "      <td>-73.968971</td>\n",
       "      <td>1</td>\n",
       "      <td>0.045132</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   taxirides__pickup_latitude  taxirides__pickup_longitude  \\\n",
       "0                   40.748283                   -73.985092   \n",
       "1                   40.722446                   -73.987366   \n",
       "2                   40.779644                   -73.955925   \n",
       "3                   40.752628                   -74.004257   \n",
       "4                   40.728329                   -73.999672   \n",
       "\n",
       "   taxirides__dropoff_latitude  taxirides__dropoff_longitude  \\\n",
       "0                    40.773758                    -73.870667   \n",
       "1                    40.648792                    -73.977425   \n",
       "2                    40.789238                    -73.968719   \n",
       "3                    40.747871                    -74.000458   \n",
       "4                    40.761410                    -73.968971   \n",
       "\n",
       "   taxirides__passenger_count  taxirides__euclid_dist  taxi_id  \n",
       "0                           1                0.117227        7  \n",
       "1                           1                0.074322       18  \n",
       "2                           1                0.015992        1  \n",
       "3                           2                0.006088       11  \n",
       "4                           1                0.045132        5  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_df = pd.DataFrame.from_dict(features_dict)\n",
    "predict_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>taxi_id</th>\n",
       "      <th>prediciton</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>18</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>4.999883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>16</td>\n",
       "      <td>5.999835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>10</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>14</td>\n",
       "      <td>8.499792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>8</td>\n",
       "      <td>8.499792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>17</td>\n",
       "      <td>5.999835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>12</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>6</td>\n",
       "      <td>4.999883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>15</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>9</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>4</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>13</td>\n",
       "      <td>8.999670</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    taxi_id  prediciton\n",
       "0         7    8.999670\n",
       "1        18    8.999670\n",
       "2         1    4.999883\n",
       "3        11    8.999670\n",
       "4         5    8.999670\n",
       "5        16    5.999835\n",
       "6        10    8.999670\n",
       "7         3    8.999670\n",
       "8        14    8.499792\n",
       "9         8    8.499792\n",
       "10        2    8.999670\n",
       "11       17    5.999835\n",
       "12       12    8.999670\n",
       "13        6    4.999883\n",
       "14        0    8.999670\n",
       "15       15    8.999670\n",
       "16        9    8.999670\n",
       "17        4    8.999670\n",
       "18       13    8.999670"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame.from_dict({'taxi_id': predict_df.taxi_id, \n",
    "                        'prediciton': model.predict(predict_df.drop('taxi_id', axis=1))})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2020 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-1.m46",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-1:m46"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
